Program, information processing method, information processing apparatus, and model generation method

ABSTRACT

A non-transitory computer-readable medium storing computer program code executed by a computer processor that executes an imaging process comprising: acquiring a medical image generated based on a signal detected by a catheter insertable into a body lumen; estimating a cause of an image defect by inputting the acquired medical image to a model learned to output the cause of the image defect when the medical image in which the image defect occurs is input; and outputting introduction information for introducing a countermeasure for removing the estimated cause of the image defect.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/JP2021/009230 filed on Mar. 9, 2021, which claims priority toJapanese Application No. 2020-058991 filed on Mar. 27, 2020, the entirecontent of both of which is incorporated herein by reference.

TECHNOLOGICAL FIELD

The present disclosure generally relates to a program, an informationprocessing method, an information processing apparatus, and a modelgeneration method.

BACKGROUND DISCUSSION

A medical diagnostic imaging apparatus that images an inside of a humanbody, such as an ultrasound diagnostic apparatus, an X-ray photographyapparatus, and an X-ray CT apparatus, has been widely used, and a methodfor detecting a failure, a breakage, or the like of the diagnosticimaging apparatus has been proposed. For example, Japanese PatentApplication Publication No. 2010-172434A discloses a medical imagingapparatus failure diagnosis support apparatus that compares a medicalimage obtained from a medical imaging apparatus with a typical image inwhich an abnormal phenomenon occurs due to a failure of the apparatus,and displays a corresponding case when the abnormal phenomenon occurs.

However, the disclosure according to Japanese Patent ApplicationPublication No. 2010-172434 A detects the abnormality simply by patternmatching with the typical image, and is not necessarily accurate.

SUMMARY

A non-transitory computer-readable program is disclosed that is capableof suitably removing a cause of an image defect occurring in a medicalimage.

A non-transitory computer-readable medium (CRM) storing computer programcode executed by a computer processor that executes a processcomprising: acquiring a medical image generated based on a signaldetected by a catheter insertable into a body lumen; estimating a causeof an image defect by inputting the acquired medical image to a modellearned to output the cause of the image defect when the medical imagein which the image defect occurs is input; and outputting introductioninformation for introducing a countermeasure for removing the estimatedcause of the image defect.

An information processing apparatus is discloses comprising: anacquisition unit configured to acquire a medical image generated basedon a signal detected by a catheter insertable into a body lumen; anestimation unit configured to estimate a cause of an image defect byinputting the acquired medical image to a model learned to output thecause of the image defect when the medical image in which the imagedefect occurs is input; and an output unit configured to outputintroduction information for introducing a countermeasure for removingthe estimated cause of the image defect.

A model generation method executed by a computer processor is disclosed,the method comprising: acquiring training data in which data indicatinga cause of an image defect is given to a medical image that is generatedbased on a signal detected by a catheter insertable into a body lumenand in which the image defect occurs; and generating, based on thetraining data, a learned model configured to output the cause of theimage defect when the medical image in which the image defect occurs isinput.

In one aspect, the cause of the image defect occurring in the medicalimage can be suitably removed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram showing a configuration example of animage diagnosis system.

FIG. 2 is a block diagram showing a configuration example of a server.

FIG. 3 is an explanatory diagram of an image defect occurring in adiagnostic imaging apparatus.

FIG. 4 is an explanatory diagram of an estimation model.

FIG. 5 is a flowchart showing a procedure for introducing an imagedefect.

FIG. 6 is an explanatory diagram showing an example of a display screenof the diagnostic imaging apparatus.

FIG. 7 is a flowchart showing a procedure for generating the estimationmodel.

FIG. 8 is a flowchart showing a procedure for estimating an imagedefect.

FIG. 9 is a block diagram showing a configuration example of a serveraccording to a second embodiment.

FIG. 10 is an explanatory diagram of artifacts.

FIG. 11 is an explanatory diagram of a detection model.

FIG. 12 is an explanatory diagram showing an example of a display screenof a diagnostic imaging apparatus according to the second embodiment.

FIG. 13 is a flowchart showing a procedure for generating the detectionmodel.

FIG. 14 is a flowchart showing a procedure for image defect estimationand artifact detection.

FIG. 15 is an explanatory diagram of a detection model according to athird embodiment.

FIG. 16 is an explanatory diagram showing an example of a display screenof a diagnostic imaging apparatus according to the third embodiment.

FIG. 17 is a flowchart showing a procedure for generating the detectionmodel according to the third embodiment.

FIG. 18 is a flowchart showing a procedure of artifact and objectdetection.

FIG. 19 is an explanatory diagram of an estimation model according to afourth embodiment.

FIG. 20 is a flowchart showing a procedure for generating the estimationmodel according to the fourth embodiment.

FIG. 21 is a flowchart showing a procedure for estimating an imagedefect according to the fourth embodiment.

DETAILED DESCRIPTION

Set forth below with reference to the accompanying drawings is adetailed description of embodiments of a program, an informationprocessing method, an information processing apparatus, and a modelgeneration method. Note that since embodiments described below arepreferred specific examples of the present disclosure, although varioustechnically preferable limitations are given, the scope of the presentdisclosure is not limited to the embodiments unless otherwise specifiedin the following descriptions.

First Embodiment

FIG. 1 is an explanatory diagram showing a configuration example of animage diagnosis system. In the present embodiment, an image diagnosissystem will be described in which presence or absence and a cause of animage defect caused by inappropriate use, breakage, failure, or the likeof a diagnostic imaging apparatus 2 are estimated based on a medicalimage acquired from the diagnostic imaging apparatus 2, and acountermeasure for removing the cause of the image defect is presentedto a user (health care worker). The image diagnosis system includes aninformation processing apparatus 1 and the diagnostic imaging apparatus2. The information processing apparatus 1 and the diagnostic imagingapparatus 2 are communicably connected to each other via a network Nsuch as a local area network (LAN) or the Internet.

The diagnostic imaging apparatus 2 is an apparatus unit for imaging aninside of a body lumen of a subject, and is an apparatus unit forperforming an ultrasound examination in a blood vessel of the subjectusing, for example, a catheter 21. The diagnostic imaging apparatus 2can include the catheter 21, a motor drive unit (MDU) 22, an imageprocessing apparatus 23, and a display apparatus 24. The catheter 21 isa medical instrument to be inserted into a blood vessel of a subject,and includes an imaging core that transmits an ultrasound based on apulse signal and receives a reflected wave from an inside of the bloodvessel. The diagnostic imaging apparatus 2 generates a tomographic image(medical image) of the inside of the blood vessel based on a signal ofthe reflected wave received by the catheter 21. The MDU 22 is a driveapparatus to which the catheter 21 is detachably attached, and controlsmotions of the imaging core of the catheter 21 in the blood vessel in alongitudinal direction and a rotational direction by driving a built-inmotor in accordance with an operation of the user. The image processingapparatus 23 is a processing apparatus that processes data of thereflected wave received by the catheter 21 to generate the tomographicimage, and includes an input interface for displaying the generatedtomographic image on the display apparatus 24 and receiving input ofvarious setting values at a time of examination.

Note that in the present embodiment, an intravascular examination isdescribed as an example. However, the body lumen to be examined is notlimited to the blood vessel, and may be, for example, an organ such as abowel. The medical image is not limited to an ultrasound image, and maybe, for example, an optical coherence tomography (OCT) image.

The information processing apparatus 1 is an information processingapparatus capable of performing various types of information processingand transmission and reception of information, and can be, for example,a server computer, or a personal computer. In the present embodiment,the information processing apparatus 1 is a server computer, and ishereinafter referred to as a server 1 for sake of simplicity. Note thatthe server 1 may be a local server installed in the same facility(hospital or the like) as the diagnostic imaging apparatus 2, or may bea cloud server communicably connected to the diagnostic imagingapparatus 2 via the Internet or the like. The server 1 functions as anestimation apparatus that estimates the presence or absence and thecause of image defect based on the medical image generated by thediagnostic imaging apparatus 2, and provides an estimation result to thediagnostic imaging apparatus 2. Specifically, as will be describedlater, the server 1 performs machine learning for learning trainingdata, and prepares in advance an estimation model 141 (see FIG. 4 ) thatoutputs the estimation result obtained by estimating the presence orabsence and the cause of image defect in the medical image using themedical image as an input. The server 1 acquires the medical image fromthe diagnostic imaging apparatus 2, inputs the medical image to theestimation model 141, and estimates the presence or absence and thecause of image defect. When it is estimated that the image defect ispresent, the server 1 outputs introduction information for introducing acountermeasure for removing the cause of the image defect to thediagnostic imaging apparatus 2 and causes the diagnostic imagingapparatus 2 to display the introduction information.

Note that in the present embodiment, the image defect is estimated inthe server 1 separate from the image diagnostic apparatus 2, and theestimation model 141 generated by the machine learning by the server 1may be installed in the image diagnostic apparatus 2 (the imageprocessing apparatus 23) and the image defect may be estimated by thediagnostic imaging apparatus 2.

FIG. 2 is a block diagram showing a configuration example of the server1. The server 1 can include a control unit 11, a main storage unit 12, acommunication unit 13, and an auxiliary storage unit 14.

The control unit 11 includes one or more arithmetic processingapparatuses such as a central processing unit (CPU), a micro-processingunit (MPU), and a graphics processing unit (GPU), and performs varioustypes of information processing, control processing, and the like byreading and executing a program P stored in the auxiliary storage unit14. The main storage unit 12 is a temporary storage area such as astatic random-access memory (SRAM), a dynamic random-access memory(DRAM), or a flash memory, and temporarily stores data necessary for thecontrol unit 11 to perform arithmetic processing. The communication unit13 is a communication module for performing processing related tocommunication, and performs transmission and reception of information toand from an outside.

The auxiliary storage unit 14 can be a nonvolatile storage area such asa large-capacity memory or a hard disk, and stores the program P whichis necessary for the control unit 11 to perform processing and otherdata. In addition, the auxiliary storage unit 14 stores the estimationmodel 141. The estimation model 141 is a machine learning model in whichthe training data is learned as described above, and is a model thatoutputs the estimation result obtained by estimating the presence orabsence and the cause of image defect using the medical image as theinput. The estimation model 141 can be used as a program moduleconstituting artificial intelligence software.

Note that the auxiliary storage unit 14 may be an external storageapparatus connected to the server 1. The server 1 may be amulti-computer including a plurality of computers, or may be a virtualmachine virtually constructed by software.

In the present embodiment, the server 1 is not limited to theabove-described configuration, and may include, for example, an inputunit that receives an operation input, a display unit that displays animage, and the like. The server 1 may include a reading unit that readsa portable storage medium 1 a such as a compact disc (CD)-ROM or adigital versatile disc (DVD)-ROM, and may read and perform the program Pfrom the portable storage medium 1 a. Alternatively, the server 1 mayread the program P from a semiconductor memory 1 b.

FIG. 3 is an explanatory diagram of the image defect occurring in thediagnostic imaging apparatus 2. The image defect to be estimated in thepresent embodiment will be described with reference to FIG. 3 .

In the medical image imaged by the diagnostic imaging apparatus 2,various image defects may occur due to the inappropriate use, thebreakage, the failure, or the like of the diagnostic imaging apparatus2. In FIG. 3 , representative image defects that occur in the diagnosticimaging apparatus 2 are shown in comparison with causes of the imagedefects.

Examples of the cause of the image defect can include trapped air,disconnection of a drive shaft inside the catheter 21, rotationinhibition of the drive shaft inside the catheter 21, connection defectbetween the catheter 21 and the MDU 22, and failure of the MDU 22. Theimage defect caused by the trapped air can be caused by air bubblesremaining in the trapped air at a distal end of the catheter 21. Whenthe air bubbles of the trapped air are not sufficiently removed bypriming before the examination, the ultrasound is attenuated by the airbubbles, and a part of the image or the whole image becomes dark. Airbubbles in a transducer at the distal end of the catheter 21 causes aphenomenon that a dark part of the image rotates in accordance withrotation of the drive shaft. Note that in FIG. 3 , for sake ofconvenience, a state in which a part of the image is dark is shown byhatching.

When the drive shaft of the catheter 21 is disconnected, the entireimage becomes dark, and ring-down in the vicinity of the center (a whitering-shaped image appearing in the vicinity of a center of the image)disappears. A sign of disconnection causes a phenomenon such as rotationof the image itself or non-uniformed rotational distortion (NURD), whichmeans a distortion of the image due to a rotation defect. There arevarious reasons for the disconnection, and for example, when thecatheter 21 is inserted into a stenosed site (a portion narrowed by aplaque) in the blood vessel, a kink (bending, twisting, crushing, or thelike) of the drive shaft occurs. Disconnection may occur when thecatheter 21 is forcibly moved back and forth in a state in which thekink of the drive shaft occurs.

When the rotation of the drive shaft of the catheter 21 is inhibited, apattern such as a mosaic pattern or a scale pattern can be generated inthe image. This phenomenon occurs due to twisting of the drive shaft,and if the drive shaft continues to be used in the twisted state, therotation is inhibited, and the image defect occurs.

Connection defect between the catheter 21 and the MDU 22 causes aphenomenon that the image becomes dark, or a radial or storm-like imageappears. In addition, failure of the MDU 22 (for example, a defect of anencoder or a deviation of a ferrite core) causes a phenomenon that theentire image becomes dark or a luminance of a part of the image (ahatched portion shown at a lower right end in FIG. 3 ) becomesrelatively high.

In the present embodiment, the server 1 can estimate the presence orabsence and the cause (type) of the image defect based on the medicalimage. Then, the server 1 outputs the introduction information forintroducing the user to take the countermeasure for removing the causeof the image defect. Note that the image defect and the cause of theimage defect are merely examples, and are not limited to the examplesthat are disclosed.

FIG. 4 is an explanatory diagram of the estimation model 141. Theestimation model 141 is the machine learning model that outputs theestimation result obtained by estimating the cause of the image defectwhen receiving the medical image in which the image defect occurs. Theserver 1 performs the machine learning for learning the predeterminedtraining data to generate the estimation model 141 in advance, acquiresthe medical image from the diagnostic imaging apparatus 2, and inputsthe medical image to the estimation model 141 to estimate the presenceor absence and the cause of image defect. The estimation model 141 willbe described with reference to FIG. 4 .

Note that as will be described later, the medical image input to theestimation model 141 may be an image during the examination in a statein which the catheter 21 is inserted into a blood vessel (body lumen) ofa subject, or may be a test image before the examination. The estimationof the image defect before and during the examination and theintroduction of the image defect before and during the examination willbe described in detail later.

The estimation model 141 can be, for example, a neural network modelgenerated by deep learning, and can be a convolutional neural network(CNN) that extracts feature data of an input image in multipleconvolution layers. The estimation model 141 can include an intermediatelayer (hidden layer) in which convolution layers in which pixelinformation on the input image is convoluted and pooling layers in whichthe convoluted pixel information is mapped are alternately connected,and extracts the feature data (feature data map) of the input image.

Note that although the estimation model 141 is described as the CNN inthe present embodiment, the estimation model 141 may be a model based onother learning algorithms such as a generative adversarial network(GAN), a recurrent neural network (RNN), a support vector machine (SVM),and a decision tree.

The server 1 performs learning using the training data in which amedical image for training is labeled with data indicating the presenceor absence of the image defect in the medical image and the cause of theimage defect if the image defect is present. Specifically, each medicalimage for training is given with a label (metadata) of “normal”indicating that the image is normal, or “air trap”, “connection defect”,“disconnection”, “rotation inhibition”, or “MDU failure” indicating thecause of the image defect. The server 1 gives the training data to theestimation model 141 to perform learning.

Note that in the present embodiment, a normal medical image is learnedas the training data. However, a medical image in which an image defectoccurs may be learned alone without normal medical images being includedin the training data. In this case, the server 1 may comprehensivelydetermine probability values of occurrence of the image defects, andestimate, for example, that the medical image is normal when theprobability values of all the image defects are equal to or less than athreshold (for example, 70% or less). Alternatively, the user mayvisually determine the presence or absence of an image defect, and whenit is determined that an image defect is present, the user may transmitthe image to the server 1 to cause the server 1 to estimate the cause ofthe image defect. In this way, the estimation model 141 only needs to beable to estimate the cause of the image defect at least when receiving amedical image in which an image defect occurs, and a configuration ofestimating the presence or absence of image defect is not essential.

The server 1 inputs the tomographic image for training to the estimationmodel 141, and acquires the estimation result obtained by estimating thepresence or absence and the cause of image defect as an output.Specifically, the probability value corresponding to each label such as“normal” or “air trap” is acquired as the output. Note that the outputfrom the estimation model 141 may not be the probability value, and maybe a value obtained by determining whether the image corresponds to eachlabel or not using a binary value (“0” or “1”).

The server 1 compares the estimation result output from the estimationmodel 141 with a correct value of the training data, and updates aparameter such as a weight between neurons such that the estimationresult and the correct value are approximate to each other. The server 1sequentially inputs the medical images for training to the estimationmodel 141 to update the parameter, and finally generates the estimationmodel 141.

In the present embodiment, the estimation model 141 receives the medicalimages (moving images) of a plurality of consecutive frames in a timeseries as the input, and estimates the presence or absence and the causeof image defect based on the medical images of the plurality of frames.Specifically, the estimation model 141 receives, as the input, themedical images of the plurality of consecutive frames along thelongitudinal direction of the blood vessel in accordance with scanningof the catheter 21. The estimation model 141 estimates the presence orabsence and the cause of image defect based on the medical images of theplurality of consecutive frames along a time axis t.

Note that in the following description, for sake of convenience, themedical images of the consecutive frames are simply referred to as“frame images”.

The server 1 may input the frame images to the estimation model 141 oneby one to perform the processing, but it is preferable to input theplurality of continuous frame images at the same time to perform theestimation. For example, the server 1 can set the estimation model 141to a 3D-CNN (for example, C3D) that handles three-dimensional inputdata. Then, the server 1 treats the frame images as three-dimensionaldata in which coordinates of the two-dimensional frame images are set astwo axes and the time t at which the frame images are acquired is set asone axis. The server 1 inputs a plurality of frame images (for example,16 frames) for a predetermined unit time as one set to the estimationmodel 141, and outputs a single estimation result (probability value ofeach label) based on the plurality of frame images. Accordingly, theestimation can be performed in consideration of the consecutivepreceding and following frame images in the time series, and anestimation accuracy can be improved.

Note that in the above description, the time series frame images can beprocessed by treating the frame images as the three-dimensional dataincluding the time axis, but the present embodiment is not limited tothe treating the frame images as the three-dimensional data includingthe time axis. For example, the server 1 may estimate the image defectbased on the plurality of consecutive frame images by using a modelobtained by combining the CNN and the RNN as the estimation model 141.In this case, for example, a long-short term memory (LSTM) layer isinserted after the intermediate layer of the CNN, and the estimation isperformed based on the feature data extracted from the plurality offrame images. Also in this case, a detection accuracy can be improved inconsideration of the plurality of consecutive frame images in the samemanner as described above.

Furthermore, in the present embodiment, as the input to the estimationmodel 141, operation information on the diagnostic imaging apparatus 2when the medical image is generated is used as the input in addition tothe medical image. The operation information can be a log indicating anoperation status of the diagnostic imaging apparatus 2 by the user, andis data capable of identifying an examination status of the subjectusing the diagnostic imaging apparatus 2.

Specifically, the server 1 determines, based on the operationinformation at the generation time point of the medical image, whetherthe time point is before the examination or during the examination (orafter the examination). Then, the server 1 inputs the determinationresult as to whether the time point is before or during the examinationto the estimation model 141 together with the medical imagecorresponding to the time point. Note that the phrase before theexamination represents a state in which the catheter 21 is not insertedinto the blood vessel of the subject (a test before the examination),and the phrase during the examination represents a state in which thecatheter 21 is inserted into the blood vessel of the subject.

For example, the server 1 can input the binary data indicating whetherthe time point is before or during the examination to the estimationmodel 141 as a categorical variable indicating an attribute of themedical image. The training data can include the operation informationas input data being correlated with the medical image, and the server 1also can input the determination result before or during the examinationdetermined based on the operation information to the estimation model141 to perform the learning.

In general, depending on whether the examination is being performed inthe diagnostic imaging apparatus 2, there are an image defect that islikely to occur during the examination and an image defect that occursregardless of whether the examination is being performed. For example,an image defect caused by disconnection or a sign of disconnection,rotation inhibition, or the like is likely to occur during theexamination, which means the catheter 21 is being operated. On the otherhand, an image defect caused by air trap, connection defect, or the likeoccurs regardless of whether the examination is being performed, andthus is relatively easy to find even before the examination. Therefore,the estimation accuracy can be improved by causing the estimation model141 to learn an examination status at a time of generating the medicalimage.

The server 1 learns the training data as described above to generate theestimation model 141. When the image defect is actually estimated, theserver 1 acquires the medical image from the diagnostic imagingapparatus 2, inputs the medical image to the estimation model 141, andestimates the presence or absence and the cause of image defect. Notethat the estimation of the image defect may be performed in real time ata time of examination, or the processing may be performed by acquiringthe recorded medical image after the examination. In the presentembodiment, for example, the processing is performed in real time at thetime of examination.

The server 1 acquires the medical image from the diagnostic imagingapparatus 2 in real time and estimates the image defect. Then, when itis estimated that the image defect is present, the server 1 outputs theestimation result of the image defect and the introduction informationfor removing the estimated cause of the image defect to the diagnosticimaging apparatus 2.

Note that in the present embodiment, it is described that an outputtarget of the introduction information is the diagnostic imagingapparatus 2, and obviously the introduction information may be output toan apparatus other than the diagnostic imaging apparatus 2 (for example,a personal computer).

The introduction information is information indicating thecountermeasure for removing the cause of the image defect, and can be,for example, a message indicating an examination method (priming or thelike) of the catheter 21, a correct operation method for avoiding thebreakage of the catheter 21, necessity of component replacement,necessity of contact to a manufacturer, or the like. In the presentembodiment, when the estimated cause of the image defect can be removedby the user per se, the server 1 outputs the introduction informationfor prompting an examination, an operation, or component replacementnecessary for removing the cause. On the other hand, when the user perse cannot remove the cause of the image defect, the server 1 outputs theintroduction information for prompting the user to contact themanufacturer or the like.

For example, when it is estimated that an image defect due to trappedair occurs, introduction information for prompting priming can beoutput. When it is estimated that connection defect occurs, introductioninformation for prompting confirmation of connection between thecatheter 21 and the MDU 22 can be output. When it is estimated thatbreakage such as disconnection or a sign of disconnection of thecatheter 21 or rotation inhibition is possible, introduction informationfor prompting correct operation for avoiding disconnection or the likeor replacement of the catheter 21 can be output. In addition, when it isestimated that a failure occurs in the MDU 22, since the MDU 22 cannotbe repaired by the user, introduction information for prompting the userto contact the manufacturer can be output.

FIG. 5 is a flowchart showing a procedure for introducing an imagedefect. FIG. 5 conceptually shows a procedure for estimating an imagedefect at the time of examination and introducing the cause of eachimage defect. The procedure for introducing an image defect will bedescribed with reference to FIG. 5 .

The server 1 acquires the operation information on the diagnosticimaging apparatus 2 at the time of examination. Then, the server 1determines whether the current status is before the examination orduring the examination based on the operation information. According towhether the current status is before the examination, the server 1branches the processing as follows.

When it is determined that the current status is before the examination,the server 1 acquires the medical image before the examination (testimage) from the diagnostic imaging apparatus 2, inputs the medical imageto the estimation model 141, and estimates the presence or absence andthe cause of image defect. Note that as described above, the server 1inputs the determination result obtained by determining whether thecurrent status is before the examination to the estimation model 141together with the medical image.

The server 1 determines whether the predetermined image defect occursbased on the estimation result output from the estimation model 141.Specifically, the server 1 determines whether an image defect due toconnection defect or trapped air occurs. Note that at a stage before theexamination, it may be determined whether an image defect due to othercauses such as disconnection occurs.

When it is determined that a predetermined image defect occurs, theserver 1 outputs the introduction information according to the cause ofthe image defect. Specifically, when it is determined that an imagedefect due to trapped air occurs, the server 1 outputs the introductioninformation for prompting the priming. When it is determined that aconnection defect between the catheter 21 and the MDU 22 occurs, theserver 1 outputs the introduction information for prompting theconfirmation of the connection between the catheter 21 and the MDU 22.

Note that when it is determined that a connection defect occurs, theserver 1 reacquires a medical image obtained after connectionconfirmation from the diagnostic imaging apparatus 2, inputs the medicalimage to the estimation model 141, and re-estimates the image defect. Asa result of the re-estimation, it can be determined whether an imagedefect due to trapped air occurs separately from the connection defect.When it can be determined that an image defect due to trapped airoccurs, the server 1 outputs the introduction information for promptingthe priming. In this way, when the server 1 outputs the introductioninformation, the server 1 reacquires a medical image obtained after thecountermeasure indicated by the introduction information is performed,re-estimates the medical image, and determines whether an image defectdue to another cause occurs. Accordingly, the user can examine thediagnostic imaging apparatus 2 in a suitable order.

When it is determined that the current status is during the examination,the server 1 acquires the medical image during the examination from thediagnostic imaging apparatus 2, inputs the medical image to theestimation model 141, and estimates the presence or absence and thecause of image defect. Then, based on the estimation result, the server1 determines whether a predetermined image defect that is likely tooccur during the examination occurs. Specifically, the server 1 candetermine whether trapped air, a disconnection or a sign ofdisconnection of the catheter 21, a rotation inhibition, a failure ofthe MDU 22, or the like occurs.

When it is determined that an image defect due to trapped air occurs,the server 1 outputs the introduction information for prompting thepriming. When it is determined that a sign of breakage of the catheter21, such as disconnection, a sign of disconnection, or rotationinhibition is present, the server 1 outputs the introduction informationfor prompting the operation for avoiding the breakage of the catheter21, the replacement of the catheter 21, acquisition of the image afterremoving the catheter 21, or the like. When it is determined that afailure occurs in the MDU 22, the server 1 outputs the introductioninformation for prompting the user to contact the manufacturer.

Note that during the examination, as in a case before the examination,the server 1 reacquires a medical image after the introductioninformation is output, and re-estimates the image defect. Specifically,as shown in FIG. 5 , when it is determined that the image defect due tothe trapped air occurs, the server 1 reacquires a medical image obtainedafter the priming is performed and re-estimates the image defect. As aresult of the re-estimation, when it is estimated that breakage ispossible, the server 1 outputs the introduction information, reacquiresa medical image obtained after the countermeasure is performed, andre-estimates the image defect. As a result of the re-estimation, when itis determined that a failure occurs in the MDU 22, the server 1 outputsthe introduction information for prompting replacement of the MDU 22.

In this way, the server 1 performs the estimation depending on whetherthe current status is before the examination or during the examination,and outputs the introduction information while repeating thereacquisition of the medical image and the re-estimation.

FIG. 6 is an explanatory diagram showing an example of a display screenof the diagnostic imaging apparatus 2. FIG. 6 shows the example of thedisplay screen in the diagnostic imaging apparatus 2 when an imagedefect occurs. FIG. 6 shows, for example, the display screen in a casein which a sign of breakage (for example, disconnection) is estimatedduring the examination.

As shown in FIG. 6 , the diagnostic imaging apparatus 2 displays amedical image (tomographic image) obtained by imaging the inside of theblood vessel of the subject. When the server 1 estimates that an imagedefect is present, the diagnostic imaging apparatus 2 displays an alertof the estimation result related to the image defect in accordance withthe output from the server 1.

The diagnostic imaging apparatus 2 displays the introduction informationfor introducing the countermeasure for removing the cause of the imagedefect. For example, when it is estimated that a sign of disconnectionof the catheter 21 is present, the diagnostic imaging apparatus 2introduces an operation method of the catheter 21 such that the userslowly pushes the catheter 21 forward while checking a display image.When the image defect is not eliminated by the operation, the diagnosticimaging apparatus 2 guides the user to replace the catheter 21.

In addition, as shown in FIG. 6 , the server 1 may generate a secondmedical image in which a feature portion of the image serving as a basisof the estimation of the image defect is visualized, and cause thediagnostic imaging apparatus 2 to display the second medical image. Thesecond medical image is an image indicating an image region referred toas the feature portion when the estimation model 141 estimates the imagedefect, and can be, for example, an image indicating the region in aheat map.

For example, the server 1 generates the second medical image usingGrad-CAM method. Grad-CAM is a method of visualizing which part of theinput image is captured as a feature in the CNN, and is a method ofextracting an image part that greatly contributes to an output. InGrad-CAM, extraction is performed by regarding a part having a largegradient when the feature data is extracted in the intermediate layer ofthe CNN as the feature portion.

Specifically, the server 1 inputs an output value (probability value ofeach label) from an output layer of the estimation model 141 (CNN) andgradient data of the input to a last convolution layer in theintermediate layer to an activation function, and generates the heatmap. The server 1 superimposes the generated heat map on an originalmedical image to generate the second medical image. As shown in a lowerright side of FIG. 6 , the server 1 displays the second medical image inparallel with the original medical image.

Note that although Grad-CAM is described above, the second medical imagemay be generated using another method such as Guided Grad-CAM. Bydisplaying the second medical image, a basis for the estimation model141 to determine the image defect can be presented to the user, and theuser can check whether the determination is correct.

FIG. 7 is a flowchart showing a procedure for generating the estimationmodel 141. With reference to FIG. 7 , a processing content when thetraining data is learned to generate the estimation model 141 will bedescribed.

The control unit 11 of the server 1 acquires the training data in whichthe data indicating the presence or absence and the cause of imagedefect in the medical image is given to the medical image for trainingand the operation information (S11). Based on the training data, thecontrol unit 11 generates the estimation model 141 that outputs theestimation result obtained by estimating the presence or absence and thecause of image defect when receiving the medical image (S12). Forexample, as described above, the control unit 11 generates a CNN modelas the estimation model 141. The control unit 11 inputs the medicalimage for training and the determination result of whether the currentstatus is before the examination, which is determined based on theoperation information, to the estimation model 141, and acquires theestimation result obtained by estimating the presence or absence and thecause of image defect as the output. The control unit 11 compares theestimation result with the correct value, and generates the estimationmodel 141 by optimizing the parameter such as a weight between neuronssuch that the estimation result and the correct value are approximate toeach other. The control unit 11 ends the series of processing.

FIG. 8 is a flowchart showing a procedure for estimating an imagedefect. With reference to FIG. 8 , a processing content when thepresence or absence and the cause of image defect is estimated using theestimation model 141 and the introduction information for removing thecause is output will be described.

The control unit 11 of the server 1 acquires the medical image from thediagnostic imaging apparatus 2 (S31). Then, the control unit 11 acquiresthe operation information when the medical image is generated from thediagnostic imaging apparatus 2 (S32).

The control unit 11 inputs the acquired medical image and thedetermination result of whether the current status is before theexamination, which is determined based on the operation information, tothe estimation model 141, and estimates the presence or absence and thecause of image defect (S33). The control unit 11 determines whether theimage defect is present based on the estimation result of S33 (S34).When it is determined that the image defect is present (YES in S34), thecontrol unit 11 generates the second medical image indicating thefeature portion in the medical image referred to when the image defectis estimated in the estimation model 141 (S35). The control unit 11outputs, to the diagnostic imaging apparatus 2, the introductioninformation for introducing the countermeasure for removing the cause ofthe image defect together with the second medical image (S36).

When it is determined to be NO in S34 or after the processing of S36 isperformed, the control unit 11 determines whether the examination by thediagnostic imaging apparatus 2 is completed (S37). When it is determinedthat the examination is not completed (NO in S37), the control unit 11returns the processing to S31. When it is determined that theexamination is completed (YES in S37), the control unit 11 ends theseries of processing.

Note that although the case of displaying the estimation result of theimage defect is displayed is described above, the server 1 may furtherreceive an input for correcting the displayed estimation result of theimage defect from the user and perform relearning based on the inputinformation. Specifically, the server 1 receives the correction inputindicating whether the image defect displayed as the estimation resultactually occurs on the display screen shown in FIG. 5 . Further, whenthe cause of the displayed image defect is different from an actualcause, the server 1 receives an input of the correct cause of the imagedefect. When the correction input is received, the server 1 performs therelearning using, as the training data, a medical image in which thecorrected estimation result (the presence or absence and the cause ofimage defect) is labeled, and updates the estimation model 141.Accordingly, the estimation accuracy of the image defect can be improvedthrough the operation of the present system.

In addition, although the estimation model 141 that is common betweenbefore and during the examination is used in the above description, theestimation model 141 obtained by learning the medical image before theexamination and the estimation model 141 obtained by learning themedical image during the examination may be separately prepared, anddifferent estimation models 141 may be used depending on whether thecurrent status is before the examination. The estimation accuracy can beimproved by preparing different models depending on whether the currentstatus is before the examination.

As described above, according to the first embodiment, by using theestimation model 141 constructed by the machine learning, the presenceor absence and the cause of image defect can be accurately estimated,and the cause of the image defect can be removed.

According to the first embodiment, the estimation accuracy can beimproved by inputting the plurality of frame images into the estimationmodel 141 and simultaneously processing the frame images.

According to the first embodiment, by repeating the reacquisition andre-estimation of the image, whether the cause of each image defectoccurs can be estimated in a suitable procedure, and the countermeasurecan be introduced to the user.

In addition, according to the first embodiment, the sign of breakage ofthe catheter 21 can be detected from the medical image, and the user canbe guided to perform a correct operation method for avoiding thebreakage or to replace the components.

Second Embodiment

In the present embodiment, in addition to estimation of an image defect,an aspect in which an artifact in a medical image is detected andpresented to a user will be described. Note that the same referencenumerals are given to the same contents as those of the firstembodiment, and description of the same reference numerals will beomitted.

FIG. 9 is a block diagram showing a configuration example of the server1 according to the second embodiment. The auxiliary storage unit 14 ofthe server 1 according to the present embodiment stores a detectionmodel 142 for artifact detection. Similarly, to the estimation model141, the detection model 142 is a machine learning model in whichtraining data is learned, and is a model that receives the medical imageas an input and outputs a detection result obtained by detecting animage region corresponding to the artifact in the medical image. Thedetection model 142 is assumed to be used as a program module thatfunctions as a part of artificial intelligence software.

Note that in the following description, for sake of convenience, theimage region corresponding to the artifact in the medical image isreferred to as an “artifact region”.

FIG. 10 is an explanatory diagram of the artifact. FIG. 10 conceptuallyshows five types of artifacts occurring in the medical image.

The artifact is a virtual image that is not intended for an examinationor that is not actually present, and is an image that is imaged due toan apparatus, an imaging condition, or the like. As shown in FIG. 10 ,examples of the artifact can include a multiple reflection (echo), aring-down, an acoustic shadow, a side lobe, an NURD, and the like.

The multiple reflection is a virtual image generated by an ultrasoundtransmitted from the catheter 21 being reflected many times in a bodylumen. The example of FIG. 10 shows a state in which the ultrasound isreflected by an object M1 (for example, a calcified tissue) in a bloodvessel and an artifact A1 is generated. When a hard object M1 is presentin the blood vessel, the artifact A1 is generated at a position at anequal interval to a distance between the object M1 and the catheter 21,and is projected as an image that is extremely similar to the object M1.

The ring-down is a ring-shaped image that appears near a center of animage due to the multiple reflection between an oscillator and a sheath.The example of FIG. 10 shows a state in which a ring-shaped artifact A2appears at the center of the image. The ring-down is projected as awhite ring having a constant width.

The acoustic shadow is a phenomenon that a part of the image fades awayin black as the ultrasound is greatly attenuated in a process of beingtransmitted radially outward of the catheter 21. The example of FIG. 10conceptually shows a state in which a region radially outward of thecatheter 21 from the object M2 fades away in black as an artifact A3.Note that in FIG. 10 , for convenience of illustration, the region thatfades away in black is indicated by hatching. When a hard object M2 ispresent in the blood vessel, most of the ultrasounds are reflected bythe object M2, and thus the ultrasound transmitted radially outward ofthe catheter 21 from the object M2 is greatly attenuated, so that theacoustic shadow is generated.

The side lobe is a weak ultrasound (sub pole) transmitted at a constantangle with respect to a main lobe (main pole) of the ultrasoundtransmitted with a constant directivity. Due to the side lobe, objectsM3 (for example, a stent) in an actual blood vessel are projected asimages larger than actual images. The example of FIG. 10 shows imagescaused by the side lobe as artifacts A4. When the catheter 21simultaneously receives reflected waves from the objects M3 on the sidelobe and a reflected wave from the main lobe, artifacts A4 aregenerated.

The NURD is a distortion of an image generated when a drive shaft of thecatheter 21 does not normally rotate. The NURD occurs due to bending inthe blood vessel, twisting of a shaft of the catheter 21, or the like.The example of FIG. 10 shows a part in which an image of a left half isdistorted due to unevenness of a rotation speed as an artifact A5surrounded by a broken line.

In the present embodiment, the server 1 detects the above-describedvarious artifacts from the medical image. Specifically, as will bedescribed below, the server 1 detects the artifact using the detectionmodel 142 which has learned the artifact region in the medical image.

Note that the multiple reflection, the ring-down, and the like areexamples of the artifacts, and the artifacts to be detected are notlimited to the examples disclosed.

FIG. 11 is an explanatory diagram of the detection model 142. Thedetection model 142 is a machine learning model that receives a medicalimage as input and outputs a detection result obtained by detecting anartifact region in the medical image. The server 1 learns the trainingdata similarly to the estimation model 141 and generates the detectionmodel 142 in advance. Then, the server 1 inputs the medical imageacquired from the diagnostic imaging apparatus 2 to the detection model142, and detects the artifact region in the medical image.

The detection model 142 can be, for example, a CNN, which includes anintermediate layer (hidden layer) in which convolution layers andpooling layers are alternately connected, and extracts feature data(feature data map) of an input image. Note that the detection model 142is described as the CNN in the present embodiment, but may be a modelbased on other learning algorithms such as a GAN, an RNN, an SVM, and adecision tree.

In the present embodiment, the server 1 generates the detection model142 for identifying, in units of pixels, whether each pixel in the inputmedical image is a pixel corresponding to the artifact region. Forexample, the server 1 generates a semantic segmentation model, a MASKregion CNN (R-CNN), or the like as the detection model 142.

The semantic segmentation model is one type of the CNN, and is one typeof an encoder decoder model that generates output data based on inputdata. The semantic segmentation model includes, in addition to theconvolution layer for compressing data of the input image, adeconvolution layer for mapping (enlarging) feature data obtained bycompression to an original image size. The deconvolution layeridentifies which object is present at which position in the image basedon the feature data extracted by the convolution layer, and generates alabel image in which each pixel corresponds to which object isbinarized.

The MASK R-CNN is a modification of Faster R-CNN mainly used for objectdetection, and has a configuration in which the deconvolution layer isconnected to the Faster R-CNN. The MASK R-CNN inputs feature data of animage extracted by the CNN and information on a coordinate region of atarget object extracted by a region proposal network (RPN) to thedeconvolution layer, and finally generates a mask image obtained bymasking the coordinate region of the object in the input image.

The server 1 generates these models as the detection model 142 and usesthese models to detect the artifact. Note that the above-describedmodels are merely examples, and the detection model 142 may be any modelas long as it can identify a position and a shape of the artifact in themedical image. In the present embodiment, for example, the detectionmodel 142 will be described as the semantic segmentation model.

The server 1 performs learning using the training data in which amedical image for training is labeled with data indicating the artifactregion. Specifically, in the training data, a label indicating acoordinate range corresponding to the artifact region and a type of theartifact is given to the medical image for training.

The server 1 inputs the medical image for training to the detectionmodel 142, and acquires a detection result obtained by detecting theartifact region as an output. Specifically, as shown by hatching on aright side of the detection model 142 in FIG. 11 , a label image inwhich data indicating the type of the artifact is labeled is acquired asan output for each pixel corresponding to the artifact region.

The server 1 compares the detection result output from the detectionmodel 142 with the coordinate range of the artifact region of thecorrect answer indicated by the training data and the type of artifact,and generates the detection model 142 by optimizing a parameter such asa weight between neurons such that the detection result and the correctanswer are approximate to each other.

Note that as in the estimation model 141, it is preferable that thedetection model 142 can perform estimation from a plurality of frameimages that are continuous in time series. In this case, similarly tothe estimation model 141, the detection model 142 may be a 3D-CNN (forexample, 3D U-net) or a model obtained by combining the CNN and the RNN.

The server 1 learns the training data as described above and generatesthe detection model 142. When the medical image is acquired from thediagnostic imaging apparatus 2, the server 1 estimates an image defectusing the estimation model 141, and inputs the medical image to thedetection model 142 to detect the artifact region. When the server 1detects the artifact region, the server 1 outputs the detection resultto the diagnostic imaging apparatus 2.

FIG. 12 is an explanatory diagram showing an example of a display screenof the diagnostic imaging apparatus 2 according to the secondembodiment. FIG. 12 shows an example of the display screen displayed bythe diagnostic imaging apparatus 2 when an artifact is detected.

When an artifact region is detected by the server 1, the diagnosticimaging apparatus 2 displays the artifact region in association with themedical image. Specifically, as indicated by the hatching in FIG. 12 ,the diagnostic imaging apparatus 2 displays a third medical imageindicating the detected artifact region in a display mode (for example,color display) different from that of the other image regions.

The third medical image can be a medical image obtained by processingthe artifact region so as to be distinguishable from the other regions,and is an image obtained by superimposing the label image output fromthe detection model 142 on an original medical image. When the artifactregion is detected, the server 1 generates the third medical image andoutputs the third medical image to the diagnostic imaging apparatus 2.For example, the server 1 processes the label image into a translucentmask of a display color other than black and white, and generates thethird medical image by superimposing the translucent mask on theartifact region of the medical image expressed in black and white.

In this case, the server 1 preferably changes the display mode (displaycolor) according to the type of the artifact. Accordingly, the user canintuitively grasp various artifacts generated due to different causes,and convenience can be improved.

Note that although the artifact region is displayed in color in theabove description, the present embodiment is not limited to the artifactregion being displayed in color, and for example, a contour (edge) partof the artifact region may be highlighted. In this way, the display modeof the artifact region is not particularly limited as long as theartifact region can be displayed so as to be distinguishable from theother image regions.

The diagnostic imaging apparatus 2 displays the third medical image andnotifies the user that the artifact occurs. A label name indicating thetype of the artifact is displayed correlated with the display color (forexample, a type of hatching in FIG. 12 ) of the artifact region.

Note that in the above description, the artifact region is detected inunits of pixels, and the artifact region can be displayed in units ofpixels, however, the present embodiment is not limited the artifactregion being detected in units of pixels. For example, the artifactregion may be simply surrounded by a bounding box (rectangular frame)and displayed. In this way, a configuration of detecting the artifactregion in units of pixels is not essential, and any configuration may beused as long as a position corresponding to the artifact can be detectedand displayed.

The example of FIG. 12 shows a case in which only the artifact isdetected, and in a case in which an image defect is also estimated(detected) at the same time, the detection result of the artifact and anestimation basis of the image defect may be displayed on the medicalimage at the same time. In this case, for example, the server 1 maydisplay, in a superimposed manner, the translucent mask corresponding tothe artifact region and a heat map corresponding to the image defect onthe same medical image.

FIG. 13 is a flowchart showing a procedure for generating the detectionmodel 142. With reference to FIG. 13 , a processing content when thedetection model 142 is generated by machine learning will be described.

The control unit 11 of the server 1 acquires the training data in whichthe medical image for training is labeled with the artifact region(S201). Specifically, as described above, the training data in which alabel indicating the coordinate range of the artifact region and thetype of the artifact is given to the medical image for training isacquired.

The control unit 11 generates, based on the training data, the detectionmodel 142 that outputs the detection result obtained by detecting theartifact region and the type of the artifact when receiving the medicalimage (S202). Specifically, as described above, the control unit 11generates, as the detection model 142, the semantic segmentation modelfor identifying an object in the medical image in units of pixels. Thecontrol unit 11 inputs the medical image for training to the detectionmodel 142, and acquires, as an output, the detection result obtained bydetecting the artifact region and the type of the artifact. The controlunit 11 compares the detection result with a correct value (a correctlabel), and generates the detection model 142 by optimizing theparameter such as a weight between neurons such that the detectionresult and the correct answer value are approximate to each other. Thecontrol unit 11 ends the series of processing.

FIG. 14 is a flowchart showing a procedure for image defect estimationand artifact detection. Note that the same steps or processes as thosein the flowchart of FIG. 8 are denoted by the same reference numerals,and description of the steps or processes will be omitted.

The control unit 11 of the server 1 performs the following processingwhen it is determined to be NO in S34 or after the processing of S36 isperformed. The control unit 11 inputs the medical image acquired fromthe diagnostic imaging apparatus 2 to the detection model 142, anddetects the artifact region (S221). Specifically, as described above,the control unit 11 detects the coordinate range of the artifact regionand the type of the artifact.

The control unit 11 determines whether the artifact region is detectedas a result of the processing at S221 (S222). When it is determined thatthe artifact region is detected (YES in S222), the control unit 11generates the third medical image in which the display mode of theartifact region is changed according to the type of the artifact (S223).The control unit 11 outputs the generated third medical image to thediagnostic imaging apparatus 2 and causes the diagnostic imagingapparatus 2 to display the third medical image (S224). When it isdetermined to be NO in S222 or after the processing of S224 isperformed, the control unit 11 shifts the processing to S37.

Note that the estimation model 141 and the detection model 142 aredescribed as separate models in the above description, but may be thesame model.

As for the detection model 142, similarly to the estimation model 141, acorrection input of the detection result may be received, and a medicalimage obtained by labeling the corrected detection result (thecoordinate range and the type of the artifact region) may be given tothe detection model 142 as the training data to perform relearning.

As described above, according to the second embodiment, not only theestimation of the image defect but also the detection of the artifactcan be performed at the same time.

Third Embodiment

The second embodiment describes an aspect in which an artifact region isdetected using the detection model 142. In the present embodiment, anaspect in which a predetermined object in a body lumen to be examined isdetected from a medical image in addition to an artifact will bedescribed.

FIG. 15 is an explanatory diagram of the detection model 142 accordingto a third embodiment. In the present embodiment, the server 1 learnstraining data in which a medical image for training is labeled with dataindicating an image region of the object to be examined (hereinafter,referred to as an “object region”) other than an artifact region, andgenerates the detection model 142. The object can be an object in ablood vessel (body lumen) to be diagnosed or treated, and can be, forexample, a plaque or the like.

Note that the object is not limited to a biological tissue present inthe blood vessel, and may be a substance other than the biologicaltissue, such as a stent indwelled in the blood vessel of a subject(patient).

In the training data, in addition to artifact data (a coordinate rangeof the artifact region and a type of an artifact) or instead of theartifact data, data relating to the object is given to the medical imagefor training. Specifically, as shown on a right side of the detectionmodel 142, the data indicating the coordinate range of the artifactregion and the type of the artifact is labeled when the artifact ispresent in the image, and data indicating a coordinate range of theobject region and a type of the object is labeled when the object ispresent.

The server 1 generates the detection model 142 based on the trainingdata. Since the present embodiment is the same as the first embodimentexcept that the object region is added, a detailed description of thedetection model 142 will be omitted in the present embodiment. When themedical image is acquired from the diagnostic imaging apparatus 2, theserver 1 inputs the medical image to the detection model 142, detectsthe artifact region and/or the object region, and outputs a detectionresult to the diagnostic imaging apparatus 2.

FIG. 16 is an explanatory diagram showing an example of a display screenof the diagnostic imaging apparatus 2 according to the third embodiment.In the present embodiment, the diagnostic imaging apparatus 2 displays athird medical image indicating the object region other than the artifactregion, and presents the third medical image to a user. When the server1 detects the artifact region and the object region at the same time,the server 1 generates the third medical image in which a display mode(display color) of each region is changed, and causes the diagnosticimaging apparatus 2 to display the third medical image. Note that forexample, the server 1 may determine a size or the like of the objectbased on a coordinate value of the object region and causes thediagnostic imaging apparatus 2 to display the determined size or thelike together with the third medical image.

FIG. 17 is a flowchart showing a procedure for generating the detectionmodel 142 according to the third embodiment.

The control unit 11 of the server 1 acquires the training data in whichthe medical image for training is labeled with the data related to theartifact region and/or the object region (S301). Based on the trainingdata, the control unit 11 generates the detection model 142 that detectsthe artifact region and/or the object region when receiving the medicalimage (S302). The control unit 11 ends the series of processing.

FIG. 18 is a flowchart showing a procedure of artifact and objectdetection. Note that the same steps or processes as those in theflowchart of FIG. 14 are denoted by the same reference numerals, anddescription of the same steps or processes as shown in FIG. 14 will beomitted.

The control unit 11 of the server 1 performs the following processingwhen it is determined to be NO in S34 or after the processing of S36 isperformed. The control unit 11 inputs the medical image acquired fromthe diagnostic imaging apparatus 2 to the detection model 142, anddetects the artifact region and/or the object region in the medicalimage (S321).

The control unit 11 determines whether the artifact region and/or theobject region is detected at S321 (S322). When it is determined that noartifact region and/or no object region is detected (NO in S322), theprocessing proceeds to S37.

When it is determined that the artifact region and/or the object regionis detected (YES in S322), the control unit 11 generates the thirdmedical image obtained by processing the artifact region and/or theobject region (S323). The control unit 11 outputs the generated thirdmedical image to the diagnostic imaging apparatus 2 and causes thediagnostic imaging apparatus 2 to display the introduction (S324). Thecontrol unit 11 shifts the processing to S37.

Note that although the artifact and the object are detected in the samedetection model 142 in the above description, models for detecting theartifact and the object may be separately provided.

As described above, according to the third embodiment, the artifact andthe object can be simultaneously detected from the medical image andpresented to the user, and a desired object or an artifact can beidentified.

Fourth Embodiment

The present embodiment describes that an image defect is estimated usinga fluoroscopic image of a body lumen of a subject in addition to amedical image (tomographic image) generated based on a signal detectedby a catheter.

FIG. 19 is an explanatory diagram of the estimation model 141 accordingto a fourth embodiment. FIG. 19 conceptually shows a state in which, inaddition to the medical image which is imaged by the diagnostic imagingapparatus 2 and operation information, the fluoroscopic image of thebody lumen of the subject is input to the estimation model 141 toestimate a cause of the image defect. With reference to FIG. 19 , anoutline of the present embodiment will be described.

The fluoroscopic image is an image obtained by visualizing the bodylumen of the subject by a method such as X-ray imaging, and is, forexample, an angiogram generated by an angiography apparatus (not shown).Note that the fluoroscopic image is not limited to an angiogram as longas it is an image by which a user can identify the body lumen of thesubject and the catheter 21 inserted into the body lumen.

When the subject receives an ultrasound examination by the catheter 21,the angiography is simultaneously performed by an angiography apparatus.For example, an X-ray opaque marker is attached to a distal end of thecatheter 21, and an insertion position (a position of the distal end) ofthe catheter 21 can be identified by the fluoroscopic image. Note thatin FIG. 19 , the insertion position of the catheter 21 is indicated by ablack circle.

The server 1 acquires the medical image of an inside of the body lumen(blood vessel) by the ultrasound examination from the diagnostic imagingapparatus 2, and acquires the fluoroscopic image of the body lumen fromthe angiography apparatus. The server 1 inputs both images to theestimation model 141, and estimates presence or absence and the cause ofthe image defect.

The server 1 acquires the presence or absence and the cause of imagedefect and the insertion position of the catheter 21 from the estimationmodel 141 as an output. In training data of the estimation model 141according to the present embodiment, for example, the presence orabsence and the cause of image defect and the insertion position of thecatheter 21 are regarded as one label, and the label is given to themedical image for training of the diagnostic imaging apparatus 2, theoperation information, and the fluoroscopic image of the angiographyapparatus. The server 1 inputs the medical image for training, theoperation information, and the fluoroscopic image to the estimationmodel 141, and performs learning so as to output a correct label.

Similarly to the first embodiment, the server 1 outputs introductioninformation according to the cause of the image defect estimated by theestimation model 141. Here, the server 1 outputs different pieces of theintroduction information according to the estimated insertion positionof the catheter 21 even if the cause of the image defect is the same.For example, as shown in FIG. 19 , in a case in which it is estimatedthat a sign of a breakage (disconnection, rotation inhibition, or thelike) in the catheter 21 is present, different alerts regarding aforward operation of the catheter 21 is output depending on whether theinsertion position of the catheter 21 is a stenosed site of a bloodvessel, a bent portion of the blood vessel, or the like. In this way,the server 1 outputs, as the introduction information, differentoperation methods according to the insertion position of the catheter21.

FIG. 20 is a flowchart showing a procedure for generating the estimationmodel 141 according to the fourth embodiment.

The control unit 11 of the server 1 acquires the training data in whichthe medical image for training by the diagnostic imaging apparatus 2,the operation information, and the fluoroscopic image by the angiographyapparatus are labeled with data indicating the presence or absence andcause of the image defect (S401). Specifically, as described above, thecontrol unit 11 acquires the training data in which the data indicatingthe insertion position of the catheter 21 in addition to the presence orabsence and the cause of image defect is given as the correct label.

Based on the training data, the control unit 11 generates the estimationmodel 141 that estimates the presence or absence and the cause of imagedefect when receiving the medical image generated using a catheter andthe fluoroscopic image at a generation time point of the medical image(S402). Specifically, as described above, the control unit 11 generatesthe estimation model 141 that outputs the insertion position of thecatheter 21 as an estimation result in addition to the presence orabsence and the cause of image defect. The control unit 11 ends theseries of processing.

FIG. 21 is a flowchart showing a procedure for estimating the imagedefect according to the fourth embodiment. The same steps or processesas those in the flowchart of FIG. 8 are denoted by the same referencenumerals, and description of the same steps or processes in FIG. 8 willbe omitted.

After the processing of S32 is performed, the control unit 11 of theserver 1 performs the following processing. The control unit 11 acquiresthe fluoroscopic image of the body lumen of the subject from theangiography apparatus (S421). Then, the control unit 11 inputs thefluoroscopic image by the angiography apparatus to the estimation model141 in addition to the medical image by the diagnostic imaging apparatus2 and the operation information, and estimates the presence or absenceand the cause of image defect and the insertion position of the catheter21 (S422). The control unit 11 shifts the processing to S34.

After the processing of S35 is performed, the control unit 11 outputsthe introduction information for removing the cause of the image defect(S423). Specifically, the control unit 11 outputs the introductioninformation according to the estimated cause of the image defect and theestimated insertion position of the catheter 21. For example, thecontrol unit 11 outputs, as the introduction information, an alertrelated to different operations of the catheter 21 according to theinsertion position. The control unit 11 shifts the processing to S37.

As described above, according to the fourth embodiment, a countermeasurecan be suitably introduced in consideration of the insertion position ofthe catheter 21 by inputting the fluoroscopic image into the estimationmodel 141.

The detailed description above describes embodiments of a program, aninformation processing method, an information processing apparatus, anda model generation method. The invention is not limited, however, to theprecise embodiments and variations described. Various changes,modifications and equivalents may occur to one skilled in the artwithout departing from the spirit and scope of the invention as definedin the accompanying claims. It is expressly intended that all suchchanges, modifications and equivalents which fall within the scope ofthe claims are embraced by the claims.

What is claimed is:
 1. A non-transitory computer-readable medium (CRM)storing computer program code executed by a computer processor thatexecutes a process comprising: acquiring a medical image generated basedon a signal detected by a catheter insertable into a body lumen;estimating a cause of an image defect by inputting the acquired medicalimage to a model learned to output the cause of the image defect whenthe medical image in which the image defect occurs is input; andoutputting introduction information for introducing a countermeasure forremoving the estimated cause of the image defect.
 2. Thecomputer-readable medium according to claim 1, further comprising:acquiring a plurality of the medical images generated along alongitudinal direction of the body lumen; and estimating the cause ofthe image defect by inputting the plurality of medical images to themodel.
 3. The computer-readable medium according to claim 1, furthercomprising: acquiring operation information on a diagnostic imagingapparatus to which the catheter is connected; determining whether ageneration time point of the medical image is before an examinationbased on the operation information; and estimating the cause of theimage defect by inputting, to the model, the medical image and adetermination result as to whether the generation time point of themedical image is before the examination.
 4. The computer-readable mediumaccording to claim 1, further comprising: reacquiring a medical imageobtained after the countermeasure is performed when the introductioninformation is output; and performing re-estimation by inputting thereacquired medical image to the model.
 5. The computer-readable mediumaccording to claim 1, further comprising: generating the acquiredmedical image in a state in which the catheter is inserted into the bodylumen; and detecting a sign of a breakage of the catheter by inputtingthe acquired medical image to the model.
 6. The computer-readable mediumaccording to claim 5, wherein, when the sign of the breakage isdetected, further comprising: outputting the output introductioninformation indicating an operation method of the catheter for avoidingthe breakage.
 7. The computer-readable medium according to claim 5,wherein, when the sign of the breakage is detected, further comprising:outputting the output introduction information prompting replacement ofthe catheter.
 8. The computer-readable medium according to claim 1,further comprising: generating the medical image in a state in which thecatheter is inserted into the body lumen, and acquiring a fluoroscopicimage of the body lumen at the generation time point of the medicalimage; and outputting the introduction information by estimating thecause of the image defect based on the medical image and thefluoroscopic image.
 9. The computer-readable medium according to claim8, further comprising: estimating an insertion position of the catheterbased on the medical image and the fluoroscopic image; and outputtingthe output introduction information indicating the operation method ofthe catheter in accordance with the estimated cause of the image defectand the estimated insertion position.
 10. The computer-readable mediumaccording to claim 1, further comprising: receiving a correction inputfor correcting an estimation result based on the model after theintroduction information is output; and updating the model by performingrelearning based on the estimated medical image and an estimation resultobtained after the correction.
 11. The computer-readable mediumaccording to claim 1, further comprising: detecting an image regioncorresponding to an artifact in the medical image by inputting theacquired medical image to a model learned to output a detection resultobtained by detecting the image region corresponding to the artifactwhen the medical image is input; and outputting the detection result inassociation with the medical image.
 12. The computer-readable mediumaccording to claim 1, further comprising: detecting an image regioncorresponding to an object to be examined in the medical image byinputting the acquired medical image to a model learned to output adetection result obtained by detecting the image region corresponding tothe object when the medical image is input; and outputting the detectionresult in association with the medical image.
 13. An informationprocessing apparatus comprising: an acquisition unit configured toacquire a medical image generated based on a signal detected by acatheter insertable into a body lumen; an estimation unit configured toestimate a cause of an image defect by inputting the acquired medicalimage to a model learned to output the cause of the image defect whenthe medical image in which the image defect occurs is input; and anoutput unit configured to output introduction information forintroducing a countermeasure for removing the estimated cause of theimage defect.
 14. A model generation method executed by a computerprocessor, the method comprising: acquiring training data in which dataindicating a cause of an image defect is given to a medical image thatis generated based on a signal detected by a catheter insertable into abody lumen and in which the image defect occurs; and generating, basedon the training data, a learned model configured to output the cause ofthe image defect when the medical image in which the image defect occursis input.
 15. The method according to claim 14, further comprising:acquiring a plurality of the medical images generated along alongitudinal direction of the body lumen; and estimating the cause ofthe image defect by inputting the plurality of medical images to themodel.
 16. The method according to claim 14, further comprising:acquiring operation information on a diagnostic imaging apparatus towhich the catheter is connected; determining whether a generation timepoint of the medical image is before an examination based on theoperation information; and estimating the cause of the image defect byinputting, to the model, the medical image and a determination result asto whether the generation time point of the medical image is before theexamination.
 17. The method according to claim 14, further comprising:reacquiring a medical image obtained after the countermeasure isperformed when the introduction information is output; and performingre-estimation by inputting the reacquired medical image to the model.18. The method according to claim 16, further comprising: generating theacquired medical image in a state in which the catheter is inserted intothe body lumen; and detecting a sign of a breakage of the catheter byinputting the acquired medical image to the model.
 19. The methodaccording to claim 18, further comprising: detecting the sign of thebreakage; and outputting the output introduction information indicatingan operation method of the catheter for avoiding the breakage.
 20. Themethod according to claim 16, further comprising: detecting the sign ofthe breakage; and outputting the output introduction informationprompting replacement of the catheter.